<!DOCTYPE html>
<html lang=zh-CN>
<head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="google-site-verification" content="xBT4GhYoi5qRD5tr338pgPM5OWHHIDR6mNg1a3euekI" />
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta property="og:description" content="samuel-24276的博客">
    <meta property="og:type" content="website">
    <meta name="description" content="samuel-24276的博客">
    <meta name="keyword"  content="aircloud">
    <link rel="shortcut icon" href="/myblog/img/favicon.ico">

    <title>
        
        爬虫常用库的安装 - samuel-24276的博客 | samuel-24276&#39;s Blog
        
    </title>

    <!-- Custom CSS -->
    
<link rel="stylesheet" href="/myblog/css/aircloud.css">

    
<link rel="stylesheet" href="/myblog/css/gitment.css">

    <!--<link rel="stylesheet" href="https://imsun.github.io/gitment/style/default.css">-->
    <link href="//at.alicdn.com/t/font_620856_28hi1hpxx24.css" rel="stylesheet" type="text/css">
    <!-- ga & ba script hoook -->
    <script></script>
<meta name="generator" content="Hexo 7.1.1"></head>

<body>

<div class="site-nav-toggle" id="site-nav-toggle">
    <button>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
        <span class="btn-bar"></span>
    </button>
</div>

<div class="index-about">
    <i> Like what you do, do what you like </i>
</div>

<div class="index-container">
    
    <div class="index-left">
        
<div class="nav" id="nav">
    <div class="avatar-name">
        <div class="avatar ">
            <img src="/myblog/img/avatar.jpg" />
        </div>
        <div class="name">
            <i>samuel-24276</i>
        </div>
    </div>
    <div class="contents" id="nav-content">
        <ul>
            <li >
                <a href="/myblog/">
                    <i class="iconfont icon-shouye1"></i>
                    <span>主页</span>
                </a>
            </li>
            <li >
                <a href="/myblog/tags">
                    <i class="iconfont icon-biaoqian1"></i>
                    <span>标签</span>
                </a>
            </li>
            <li >
                <a href="/myblog/archives">
                    <i class="iconfont icon-guidang2"></i>
                    <span>存档</span>
                </a>
            </li>
            <li >
                <a href="/myblog/collect/">
                    <i class="iconfont icon-shoucang1"></i>
                    <span>收藏</span>
                </a>
            </li>
            <li >
                <a href="/myblog/about/">
                    <i class="iconfont icon-guanyu2"></i>
                    <span>关于</span>
                </a>
            </li>
            
            <li>
                <a id="search">
                    <i class="iconfont icon-sousuo1"></i>
                    <span>搜索</span>
                </a>
            </li>
            
        </ul>
    </div>
    
        <div id="toc" class="toc-article">
    <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#1-%E9%AA%8C%E8%AF%81%E5%86%85%E7%BD%AE%E5%BA%93%EF%BC%9Aurllib%E3%80%81re"><span class="toc-text">1.验证内置库：urllib、re</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#2-%E5%AE%89%E8%A3%85requests"><span class="toc-text">2.安装requests</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#3-%E5%AE%89%E8%A3%85selenium"><span class="toc-text">3.安装selenium</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#4-%E5%AE%89%E8%A3%85lxml"><span class="toc-text">4.安装lxml</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#5-%E5%AE%89%E8%A3%85beautifulsoup4"><span class="toc-text">5.安装beautifulsoup4</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#6-%E5%AE%89%E8%A3%85pyquery"><span class="toc-text">6.安装pyquery</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#7-%E5%AE%89%E8%A3%85pymysql"><span class="toc-text">7.安装pymysql</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#8-%E5%AE%89%E8%A3%85pymongo"><span class="toc-text">8.安装pymongo</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#9-%E5%AE%89%E8%A3%85redis"><span class="toc-text">9.安装redis</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#10-%E5%AE%89%E8%A3%85flask"><span class="toc-text">10.安装flask</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#11-%E5%AE%89%E8%A3%85django"><span class="toc-text">11.安装django</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#12-%E5%AE%89%E8%A3%85scrapy"><span class="toc-text">12.安装scrapy</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#13-%E5%AE%89%E8%A3%85jupyter"><span class="toc-text">13.安装jupyter</span></a></li></ol>
</div>
    
</div>


<div class="search-field" id="search-field">
    <div class="search-bg" id="search-bg"></div>
    <div class="search-container">
        <div class="search-input">
            <span id="esc-search"> <i class="icon-fanhui iconfont"></i></span>
            <input id="search-input"/>
            <span id="begin-search">搜索</span>
        </div>
        <div class="search-result-container" id="search-result-container">

        </div>
    </div>
</div>

        <div class="index-about-mobile">
            <i> Like what you do, do what you like </i>
        </div>
    </div>
    
    <div class="index-middle">
        <!-- Main Content -->
        


<div class="post-container">
    <div class="post-title">
        爬虫常用库的安装
    </div>

    <div class="post-meta">
        <span class="attr">发布于：<span>2020-03-20 22:14:38</span></span>
        
        <span class="attr">标签：/
        
        <a class="tag" href="/myblog/tags/#工具" title="工具">工具</a>
        <span>/</span>
        
        <a class="tag" href="/myblog/tags/#爬虫" title="爬虫">爬虫</a>
        <span>/</span>
        
        <a class="tag" href="/myblog/tags/#Python" title="Python">Python</a>
        <span>/</span>
        
        
        </span>
        <span class="attr">访问：<span id="busuanzi_value_page_pv"></span>
</span>
</span>
    </div>
    <div class="post-content no-indent">
        <p><strong>请求库：requests、selenium<br>解析库:lxml、beautifulsoup、pyquery<br>存储库:mongodb、redis、mysql、pymsql<br>工具库：</strong></p>
<blockquote>
<p><strong>文章基于Python3、pip、MongoDB、Redis、Mysql软件已完成环境配置的条件下进行</strong></p>
</blockquote>
<blockquote>
<p>安装在普通cmd下，测试需进入python命令行，因此开两个窗口更方便</p>
</blockquote>
<h1 id="1-验证内置库：urllib、re"><a href="#1-验证内置库：urllib、re" class="headerlink" title="1.验证内置库：urllib、re"></a>1.验证内置库：urllib、re</h1><p>进入python命令行，输入：<br><code>import urllib</code><br><code>import urllib.request</code><br><code>urllib.request.urlopen(&#39;http://baidu.com&#39;)</code><br><code>import re</code><br>若无报错信息，证明内置库已安装</p>
<h1 id="2-安装requests"><a href="#2-安装requests" class="headerlink" title="2.安装requests"></a>2.安装requests</h1><p>验证pip能否使用（CMD）：<code>pip</code>,出现参数列表，证明可以使用该命令<br>若不能使用，请去配置环境变量path</p>
<p><code>pip install requests -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import requests</code><br><code>requests.get(&#39;http://baidu.com&#39;)</code><br>若出现Response [200]&gt;的返回，证明该库已安装</p>
<h1 id="3-安装selenium"><a href="#3-安装selenium" class="headerlink" title="3.安装selenium"></a>3.安装selenium</h1><p>驱动浏览器，用来做自动化测试</p>
<p><code>pip install selenium -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import selenium</code><br><code>from selenium import webdriver</code><br><code>driver = webdriver.Chrome()</code><br>此时会打开浏览器，若不能打开，提示找不到文件，还需下载webdriver。到官网下载，不要选最新版本（最新版本只支持Chrome v81），下载完成后将解压的.exe文件复制到python安装的根目录和scripts目录下即可。<br>之后将以上代码再输入一遍，并补充：<br><code>driver.get(&#39;http://baidu.com&#39;)</code><br>此时你的浏览器便会跳转到输入的网址上，再输入：<br><code>driver.page_source</code><br>便可以打印出该网页的源代码了</p>
<h1 id="4-安装lxml"><a href="#4-安装lxml" class="headerlink" title="4.安装lxml"></a>4.安装lxml</h1><p><code>pip install lxml -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<h1 id="5-安装beautifulsoup4"><a href="#5-安装beautifulsoup4" class="headerlink" title="5.安装beautifulsoup4"></a>5.安装beautifulsoup4</h1><p><code>pip install beautifulsoup4 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>from bs4 import BeautifulSoup</code><br><code>soup = BeautifulSoup(&#39;&lt;html&gt;&lt;/html&gt;&#39;, &#39;lxml&#39;)</code></p>
<h1 id="6-安装pyquery"><a href="#6-安装pyquery" class="headerlink" title="6.安装pyquery"></a>6.安装pyquery</h1><p>与beautifulsoup4功能类似，与Java的JQuery类似</p>
<p><code>pip install pyquery -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>from pyquery import PyQuery as pq</code><br><code>doc = pq(&#39;&lt;html&gt;hello&lt;/html&gt;&#39;)</code><br><code>result = doc(&#39;html&#39;).text()</code></p>
<h1 id="7-安装pymysql"><a href="#7-安装pymysql" class="headerlink" title="7.安装pymysql"></a>7.安装pymysql</h1><p><code>pip install pymysql -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import pymysql</code><br><code>conn = pymysql.connect(host=&#39;localhost&#39;, user=&#39;root&#39;, password=&#39;root&#39;,port=3306, db = &#39;mysql&#39;)</code>密码是安装sql时设置的<br><code>cursor = conn.cursor()</code><br><code>cursor.execute(&#39;select * from db&#39;)</code><br><code>cursor.fetchone()</code></p>
<h1 id="8-安装pymongo"><a href="#8-安装pymongo" class="headerlink" title="8.安装pymongo"></a>8.安装pymongo</h1><p>非关系型数据库</p>
<p><code>pip install pymongo -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code><br>测试</p>
<p><code>import pymongo</code><br><code>client = pymongo.MongoClient(&#39;localhost&#39;)</code><br><code>db = client[&#39;newtestdb&#39;]</code><br><code>db[&#39;family&#39;].insert_one(&#123;&#39;father&#39;:&#39;bob&#39;&#125;)</code><br><code>db[&#39;family&#39;].find_one(&#123;&#39;father&#39;:&#39;bob&#39;&#125;)</code></p>
<h1 id="9-安装redis"><a href="#9-安装redis" class="headerlink" title="9.安装redis"></a>9.安装redis</h1><p>非关系型数据库，用于分布式爬虫</p>
<p><code>pip install redis -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import redis</code><br><code>r=redis.Redis(&#39;localhost&#39;, 6379)</code><br><code>r.set(&#39;father&#39;, &#39;bob&#39;)</code>						&#x2F;&#x2F;出错<br><code>r.get(&#39;father&#39;)</code></p>
<h1 id="10-安装flask"><a href="#10-安装flask" class="headerlink" title="10.安装flask"></a>10.安装flask</h1><p>web库</p>
<p><code>pip install flask -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import flask</code></p>
<h1 id="11-安装django"><a href="#11-安装django" class="headerlink" title="11.安装django"></a>11.安装django</h1><p>web服务器框架</p>
<p><code>pip install django -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import django</code></p>
<h1 id="12-安装scrapy"><a href="#12-安装scrapy" class="headerlink" title="12.安装scrapy"></a>12.安装scrapy</h1><p>用 Python 实现的一个为了爬取网站数据、提取结构性数据而编写的应用框架。<br>Scrapy 常应用在包括数据挖掘，信息处理或存储历史数据等一系列的程序中。<br>通常我们可以很简单的通过 Scrapy 框架实现一个爬虫，抓取指定网站的内容或图片。</p>
<p><code>pip install scrapy -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试<br><code>import scrapy</code></p>
<h1 id="13-安装jupyter"><a href="#13-安装jupyter" class="headerlink" title="13.安装jupyter"></a>13.安装jupyter</h1><p>网页记事本</p>
<p><code>pip install jupyter -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com</code></p>
<p>测试（CMD中）<br><code>jupyter notebook</code></p>
<hr>

        
            <div class="donate-container">
    <div class="donate-button">
        <button id="donate-button">赞赏</button>
    </div>
    <div class="donate-img-container hide" id="donate-img-container">
        <img id="donate-img" src="" data-src="/myblog/img/donate.jpg">
        <p> 感谢鼓励 </p>
    </div>
</div>
        
        <br />
        <div id="comment-container">
        </div>
        <div id="disqus_thread"></div>
        <div id="lv-container"></div>
        <div class="giscus"></div>
    </div>
</div>

    </div>
</div>


<footer class="footer">
    <ul class="list-inline text-center">
        
        

        

        

        
        <li>
            <a target="_blank"  href="https://github.com/samuel-24276">
                            <span class="fa-stack fa-lg">
                                <i class="iconfont icon-github"></i>
                            </span>
            </a>
        </li>
        

        

    </ul>
    
    <p>
        <span>/</span>
        
        <span><a target="_blank" rel="noopener" href="https://github.com/samuel-24276">samuel-24276&#39;s Page</a></span>
        <span>/</span>
        
        <span><a target="_blank" rel="noopener" href="https://beian.miit.gov.cn/">北京-未备案</a></span>
        <span>/</span>
        
        <span><a href="#">返回顶部</a></span>
        <span>/</span>
        
    </p>
    
    <p>
        <span id="busuanzi_container_site_pv">
            <span id="busuanzi_value_site_pv"></span>PV
        </span>
        <span id="busuanzi_container_site_uv">
            <span id="busuanzi_value_site_uv"></span>UV
        </span>
        Created By <a target="_blank" rel="noopener" href="https://hexo.io/">Hexo</a>  Theme <a target="_blank" rel="noopener" href="https://github.com/aircloud/hexo-theme-aircloud">AirCloud</a></p>
</footer>




</body>

<script>
    // We expose some of the variables needed by the front end
    window.hexo_search_path = "search.json"
    window.hexo_root = "/myblog/"
    window.isPost = true
</script>
<script src="https://cdn.bootcss.com/jquery/3.3.1/jquery.min.js"></script>

<script src="/myblog/js/index.js"></script>

<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>






    <script type="text/javascript">
       (function() {
           if (typeof LivereTower === 'function') { return; }

           var j, d = document.getElementById('lv-container');

           d.setAttribute('data-id','city');
           d.setAttribute('data-uid' , 'MTAyMC80OTA5OC8yNTU5Mw==');

           j = document.createElement('script');
           j.src = 'https://cdn-city.livere.com/js/embed.dist.js';
           j.async = true;

           d.appendChild(j);
       })();
    </script>
    <noscript>为正常使用来必力评论功能请激活JavaScript</noscript>
    </div>

</html>
